<html>
<head><meta charset="utf-8"><title>Generate mbe benchmark tests · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Generate.20mbe.20benchmark.20tests.html">Generate mbe benchmark tests</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="225429489"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Generate%20mbe%20benchmark%20tests/near/225429489" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Generate.20mbe.20benchmark.20tests.html#225429489">(Feb 06 2021 at 20:47)</a>:</h4>
<p>It is quite hard to generate mbe benchmark tests which is  self contain: it means we cannot use any name resolution facility. I can find all <code>ast::macro_rules</code> and <code>ast::macro_call</code> inside RA project, but that's not that much (around 2k calls). <br>
The total macro invocations in <code>rust-analyzer analysis-stats .</code> is around 60k which include std and all dependencies, and recursive calls.</p>
<p>If we want to benchmark the real world sitution, maybe only 2k calls are not enough IMO. On the other hand, that 2k call data size is around 100kb, we maybe can just keep it inside git, right?</p>
<p>I feel stuck in both cases, <span class="user-mention" data-user-id="133169">@matklad</span>  what do you think about it ?</p>



<a name="225455959"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Generate%20mbe%20benchmark%20tests/near/225455959" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Generate.20mbe.20benchmark.20tests.html#225455959">(Feb 07 2021 at 10:08)</a>:</h4>
<blockquote>
<p>If we want to benchmark the real world sitution, maybe only 2k calls are not enough IMO. On the other hand, that 2k call data size is around 100kb, we maybe can just keep it inside git, right?</p>
</blockquote>
<p>How long does it take to run for those?</p>



<a name="225455985"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Generate%20mbe%20benchmark%20tests/near/225455985" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Generate.20mbe.20benchmark.20tests.html#225455985">(Feb 07 2021 at 10:09)</a>:</h4>
<p>Worst case, we can store the fixture in another repo, ignore the test, and clone it on-demand (on CI?). But I'm not sure it's worth the hassle?</p>



<a name="225460284"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Generate%20mbe%20benchmark%20tests/near/225460284" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Generate.20mbe.20benchmark.20tests.html#225460284">(Feb 07 2021 at 11:48)</a>:</h4>
<blockquote>
<p>How long does it take to run for those?</p>
</blockquote>
<p>For 60k one, it is around 15s in my low spec machine. I haven't testetd the 2k one yet, but I guess it is proportional to the 60k one.</p>



<a name="225460620"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Generate%20mbe%20benchmark%20tests/near/225460620" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Generate.20mbe.20benchmark.20tests.html#225460620">(Feb 07 2021 at 11:56)</a>:</h4>
<blockquote>
<p>Worst case, we can store the fixture in another repo, ignore the test, and clone it on-demand (on CI?). But I'm not sure it's worth the hassle?</p>
</blockquote>
<p>Another workaround is adding new argument flags (e.g. dump-macro-rules, dump-macro-call, etc) in <code>rust-analyzer</code> cli and generate these test on CI. But I don't know whether it's worth or not.</p>



<a name="225460638"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Generate%20mbe%20benchmark%20tests/near/225460638" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Generate.20mbe.20benchmark.20tests.html#225460638">(Feb 07 2021 at 11:57)</a>:</h4>
<p>But I can imagine we can reuse these infrastructure to benchmark different part of RA , right ?</p>



<a name="225665028"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Generate%20mbe%20benchmark%20tests/near/225665028" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Generate.20mbe.20benchmark.20tests.html#225665028">(Feb 09 2021 at 09:50)</a>:</h4>
<p>Okay, I think I have an idea : We could just use a pre-generated macro-rules (which is not that big, around 640kb)  and generated macro invocations based on these rules. I will try to implement it this week to see how it is.</p>



<a name="225694093"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Generate%20mbe%20benchmark%20tests/near/225694093" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Generate.20mbe.20benchmark.20tests.html#225694093">(Feb 09 2021 at 14:07)</a>:</h4>
<p>I don't think we necessary should target "real world". I think it's better to have generated synthetic benchmarks. That way, we can easily change the generator to exercise particularly problematic spots</p>



<a name="225694239"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/Generate%20mbe%20benchmark%20tests/near/225694239" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/Generate.20mbe.20benchmark.20tests.html#225694239">(Feb 09 2021 at 14:08)</a>:</h4>
<p>Let me maybe do this for accidentally quadratic benchmark, just to get the infra in place</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>